Expert opinion crowdsourcing

ABSTRACT

An expert opinion crowdsourcing system is disclosed that may enable a person seeking an opinion (or other work product) to efficiently access experts (or other persons) who may provide such opinions (or other work products). For example, the system may enable a person to submit a request to the system, at which point the system may automatically match the request to one or more appropriate experts. The system may then provide the request to the appropriate experts, and receive opinions back from the experts in response to the request. The opinions may then be provided back to the person that submitted the request. The request may include various characteristics and/or criteria that may be matched to, or satisfied by, other characteristics or criteria associated with the experts. The system may include aspects whereby requests and/or opinions may be anonymized and/or combined.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of priority under 35 U.S.C. § 119(e)of U.S. Provisional Application No. 61/780,640, filed Mar. 13, 2013, thedisclosure of which is hereby incorporated by reference herein in itsentirety.

BACKGROUND

In medicine and other fields people may desire opinions from, or otherwork performed by, other persons, such as experts. For example, apatient or referring doctor may desire a medical opinion or evaluationby a specialist, another doctor, and/or any other type of expert. Inanother example, a person seeking legal counsel may desire an opinionfrom an expert such as an attorney.

SUMMARY

The systems, methods, and devices described herein each have severalaspects, no single one of which is solely responsible for its desirableattributes. Without limiting the scope of this disclosure, severalnon-limiting features will now be described briefly.

According to an embodiment, a computer-implemented method is disclosedcomprising: under direction of one or more hardware processorsconfigured with specific software instructions, receiving a medicalimage series including one or more medical images; providing a userinterface to a user, the user interface configured to allow the user toset preferences for selection of one or more reviewers, the preferencesincluding rules indicating preferences regarding: whether reviewersoffer availability to be contacted directly by the user; whetherreviewers offer availability to review the medical image series as partof at least one of: a legal investigation, an insurance investigation, aconsultation with a doctor, or a request of a patient; a minimum and/ormaximum quantity of reviewers to be selected to review the medical imageseries; a minimum and/or maximum quantity of reviewers permitted toprovide review information; and/or a minimum average user feedbackrequired for reviewers to be selected for review of the medical imageseries; determining, based on the preferences set by the user, one ormore reviewers to review the medical image series; and providing anotice to the determined one or more reviewers indicating availabilityof the medical image series for review.

According to another embodiment, a computing system is disclosedcomprising: a storage device configured to store electronic softwareinstructions; and one or more computer processors configured to executethe stored software instructions to cause the computing system to:receive a case including associated case characteristics, the caseprovided by a submitter, the case characteristics including a submittertype; receive expert information associated with experts, the expertinformation including expert characteristics associated with eachrespective expert, the expert characteristics including indications ofaccepted submitter types; match the case to one or more experts based onthe case characteristics and the expert characteristics, wherein thesubmitter type associated with the case matches the indications ofaccepted submitter types associated with the one or more matchedexperts; and provide the case to the matched experts.

According to yet another embodiment, a computer-readable, non transitorystorage medium is disclosed that stores computer-executable instructionsthat, when executed by a computer system, configure the computer systemto perform operations comprising: receiving a case including associatedcase characteristics, the case provided by a submitter, the casecharacteristics including an indication that the submitter desires to beable to contact an expert; receiving expert information associated withexperts, the expert information including expert characteristicsassociated with each respective expert, the expert characteristicsincluding indications whether respective experts are willing to becontacted by the submitter; matching the case to one or more expertsbased on the case characteristics and the expert characteristics,wherein case characteristics associated with the one or more matchedexperts indicated that the respective experts are willing to becontacted by the submitter; and providing the case to the matchedexperts.

BRIEF DESCRIPTION OF THE DRAWINGS

The following aspects of the disclosure will become more readilyappreciated as the same become better understood by reference to thefollowing detailed description, when taken in conjunction with theaccompanying drawings.

FIG. 1A is a flow diagram illustrating an example method of an expertopinion crowdsourcing system, according to an embodiment of the presentdisclosure.

FIG. 1B is a flow diagram illustrating another example method of anexpert opinion crowdsourcing system similar to the method of FIG. 1A,according to an embodiment of the present disclosure.

FIGS. 2A-2B are block diagrams illustrating example computing systemsand/or devices that may be included in the expert opinion crowdsourcingsystem, according to embodiments of the present disclosure.

FIG. 3 is a block diagram illustrating various examples of submittersand expert computing devices communicating with a crowdsourcing serverof the expert opinion crowdsourcing system, according to an embodimentof the present disclosure.

FIGS. 4-8 are flow diagrams illustrating example methods or processes ofthe expert opinion crowdsourcing system in which cases from submittersare provided to experts, according to embodiments of the presentdisclosure.

FIGS. 9-10 and 11A-11B are flow diagrams illustrating example methods orprocesses of the expert opinion crowdsourcing system in which cases fromsubmitters are provided to experts, and reports from experts areprovided to submitters, according to embodiments of the presentdisclosure.

DETAILED DESCRIPTION Overview

As mentioned above, in medicine and other fields people may desireopinions from, or other work performed by, other persons. For example, apatient or referring doctor may desire a medical opinion or evaluationby a specialist, another doctor, and/or any other type of expert. Inanother example, a person seeking legal counsel may desire an opinionfrom an expert such as an attorney. As various persons desire or seekopinions, counsel, and/or other work from experts, there is a need forsystems and methods that may allow such persons to efficiently accesssuch experts.

Disclosed herein, according to various embodiments, is an expert opinioncrowdsourcing system (also referred to as the “system”) that may enablea person seeking an opinion (or other work product) to efficientlyaccess experts (or other persons) who may provide such opinions (orother work products). For example, the system may enable a person tosubmit a request to the system, at which point the system mayautomatically match the request to one or more appropriate experts. Thesystem may then provide the request to the appropriate experts, andreceive opinions back from the experts in response to the request. Theopinions may then be provided back to the person that submitted therequest. In various embodiments, the request may include variouscharacteristics and/or criteria that may be matched to, or satisfied by,other characteristics or criteria associated with the experts.Additionally, the system may include aspects whereby requests and/oropinions may be anonymized and/or combined, as described below. Further,according to various embodiments, experts and/or opinions provided byexperts may be rated, as also described below. In some embodiments, aperson submitting a request to the system may provide a payment toreceive opinions.

While many of the examples and figures of the present disclosuredescribe the expert opinion crowdsourcing system in the context ofmedicine and medical image review and assessment, the systems andmethods described have equal applicability to any number of other fieldsand, thus, references herein to such medical applications may beinterpreted to cover any other field and/or applications.

Terms

In order to facilitate an understanding of the systems and methodsdiscussed herein, a number of terms are defined below. The terms definedbelow, as well as other terms used herein, should be construed toinclude the provided definitions, the ordinary and customary meaning ofthe terms, and/or any other implied meaning for the respective terms.Thus, the definitions below do not limit the meaning of these terms, butonly provide exemplary definitions.

Submitter: A person, group of persons, and/or any other type of entity,that may submit requests (for example, cases) to the expert opinioncrowdsourcing system. For example, a submitter may be a doctor thatsubmits a request for an evaluation of a medical image. In anotherexample, a submitter may be a person that submits a request for anevaluation of a legal situation. In other examples, submitters mayinclude patients, referring doctors, radiologists, insurance companies,attorneys that desire opinions from, or other work done by, experts suchas radiologists or other legal professionals, and/or any other entity.Various characteristics may be associated with, or provided by,submitters which may be utilized by the system when matching requests(for example, cases) with experts, as described below. In variousembodiments, characteristics may be referred to herein as “criteria” or“preferences.”

Expert or Reviewer: A person, group of persons, and/or any other type ofentity, that may receive requests from the expert opinion crowdsourcingsystem and provide responses to those requests. Various characteristicsmay be associated with (or provided by) experts or reviewers, which maybe matched with criteria/characteristics associated with requests. Forexample, experts/reviewers may be associated with types, specialtiesand/or sub-specialties, medical image modalities, ratings, experience,and the like. In an example, an expert may be a radiologist (or otherspecialized doctor) that may receive requests to evaluate medicalimages, and may provide evaluations and/or opinions of those medicalimages. In another example, particular types of experts, for exampleradiologists, may be associated with a sub-specialty, for example,neuroradiology, musculoskeletal, and/or the like. In yet another exampleinvolving medical imaging, experts may include cardiologists and/ornuclear medicine specialists. In a further example, an expert/reviewermay be a lawyer that may receive a request for an evaluation of a legalsituation, and may provide an evaluation or opinion regarding the legalsituation. In various embodiments, experts/reviewers may specifyparticular characteristics related to their expertise, such as types ofcases/requests that they are willing and/or unwilling to accept, and/ortypes of submitters from whom they are willing to accept cases, amongothers. For example, a doctor may indicate a specialty in evaluation ofmedical images of the brain, and/or that he accepts only requests toevaluate particular types of images of the brain (for example, MRIimages). Although the term “expert” is used in the present disclosure,any type of person, group of persons, and/or any other type of entitythat may receive requests for review of information and provideresponses to the system may fall within the scope of the presentdisclosure (whether or not considered an “expert” under common usage ofthat term).

Case: A request provided by a submitter to the expert opinioncrowdsourcing system. Cases may take any form, may be of any type, mayinclude any characteristics and/or criteria, and may be from, or applyto, any field of endeavor. Characteristics associated with a case mayinclude items that may be matched to, or satisfied by, othercharacteristics or criteria associated with experts. For example, a casemay specify a role associated with the submitter (for example, patient,doctor or lawyer), request type (for example, medical image evaluation),an anatomical area of a medical image (for example, brain), a modalityof a medical image (for example, PET, CT, MRI), a minimum ratingassociated with experts, and/or any other characteristic or criteria.

Report: A work product provided by an expert to the system (and/orsubmitter) in response to a received case. As with cases, reports maytake any form, may be of any type, may include any characteristicsand/or criteria, and may be from, or apply to, any field of endeavor. Areport may be, for example, an evaluation, an opinion, a writtenproduct, a visual product, an audio product, a tactile product, acreative product, and/or any other work product. Although the term“report” is used in the present disclosure, any type of work productproduced by an expert, in any medium, may fall within the scope of thepresent disclosure. In various embodiments, reports may be referred toherein as “reviews” and/or “review information.”

Feedback: Quantitative and/or qualitative assessment of a report, suchas an accuracy, quality, readability, format, etc. of a report. Feedbackmay also include an assessment of other characteristics of an expertthat are not directly tied to the experts report, such as timeliness inproviding the report, ease of availability of the expert, demeanor ofthe expert in dealing with the submitter, etc. Feedback, such as from asubmitter that receives a report from an expert, may be used to scoreand/or rate the expert. Such ratings may then be used to filter theexperts that are selected to review and report on a new case. Forexample, a submitter may restrict access to an uploaded case to onlyexperts having a minimum feedback rating, where feedback ratings forexperts are some aggregate (e.g., average, possibly in multiple feedbackcategories) of feedback ratings from multiple submitters. Otherindividuals or groups of individuals may provide feedback on reportsand/or experts that provided respective reports. For example, “feedbackentities,” which is any entity that provides feedback on a report, mayinclude experts, the submitter, and/or any other entity from whichfeedback on reports may be desired (e.g., individuals that are neitherthe submitter nor an expert that provides a report).

Figures

Embodiments of the disclosure will now be described with reference tothe accompanying figures, wherein like numerals refer to like elementsthroughout. The terminology used in the description presented herein isnot intended to be interpreted in any limited or restrictive manner,simply because it is being utilized in conjunction with a detaileddescription of certain specific embodiments of the disclosure.Furthermore, embodiments of the disclosure may include several novelfeatures, no single one of which is solely responsible for its desirableattributes or which is essential to practicing the embodiments of thedisclosure herein described.

Example Method

FIG. 1A is a flow diagram illustrating an example method of an expertopinion crowdsourcing system (also referred to as the “system”),according to an embodiment of the present disclosure. The method of FIG.1 A may be performed by a crowdsourcing server 100 (FIG. 2A) and/orother suitable computing device. Depending on the implementation, thesystem may perform a method having more or fewer blocks than are shown,and/or the blocks may occur in a different order and/or in parallel inorder to accomplish the methods and/or processes of the system.

Beginning at block 1205, a submitter may provide criteria and/orcharacteristics related to a case submission and/or a selection ofexperts (or other types of reviewers). In an embodiment, submitters mayindicate criteria/characteristics related to how a case may be managedby the system. For example, as mentioned above, characteristicsassociated with a case may include items that may be matched to, orsatisfied by, other characteristics or criteria associated with experts.For example, a case may specify a role associated with the submitter(for example, doctor or lawyer), request type (for example, medicalimage evaluation), an anatomical area of a medical image (for example,brain), a modality of a medical image (for example, PET, CT, MRI), aminimum rating associated with experts, and/or any other characteristic.Other examples of characteristics/criteria that may be provided by asubmitter in association with a case may include:

-   -   Qualifications for types of experts the submitter may be willing        to accept. For example, a doctor with a particular specialty, or        an expert having a particular rating.    -   Requests for specific identified experts. For example, the        submitter may provide a selection of particular experts from a        list of available experts. In an embodiment, a list of available        experts is provided by the system and may be automatically        narrowed based on criteria/characteristics provided by the        submitter (for example, characteristics mentioned above).    -   A field or fields of expertise, and/or a triage function. For        example, the submitter may direct a case to an expert with a        particular specialty, for example, neuroradiology,        musculoskeletal, GI, cardiology, or the like.    -   A number of expert reports and/or opinions desired. For example,        the submitter may desire 1, 2, 3, 4, 5, and/or more opinions. In        an example, the submitter may desire reports from multiple        different experts having different characteristics.    -   A request that a case be provided to experts who are willing to        verbally discuss the case. In an embodiment, such a request        provided to the system may require an additional charge to, or        payment by, the submitter.    -   An expert blacklist. For example, a submitter may blacklist, or        indicate that they do not want a case to be matched with,        experts with whom they may have had an unsatisfactory        interaction in the past. In another example, a submitter may        indicate that experts having a rating below a particular        threshold are to be blacklisted.    -   A request that the case be provided to an expert who is        available and/or able to complete a report in a particular        period of time, or in a particular timeframe. As mentioned        below, experts may provide, and/or the system may automatically        determine, an availability and/or an expected time to complete a        report for any particular expert. Accordingly, in an embodiment,        a submitter may indicate a desire and/or requirement that a        report be completed within a particular period of time, and the        system may match the case to experts that are available and/or        capable (and/or likely) to complete the report in the particular        period of time.

Moving to block 1210, experts may indicate and/or be associated withcriteria and/or characteristics related to cases and/or submitters theywill accept. For example, experts may indicate criteria/characteristicsrelated to how the system matches them with cases and submitters. Asmentioned above, experts may be associated with types, specialtiesand/or sub-specialties, medical image modalities, ratings, and the like.Other examples of characteristics/criteria that may be provided by,and/or be associated with, experts may include:

-   -   Specific types of exams or other work the experts are willing to        take, for example, MRI and CT of the elbow.    -   Rules indicating types of submitters from which the experts may        accept cases and/or purposes of the requested expert review for        which the experts may accept cases. For example, an expert may        indicate that they may (or may not) accept cases based on one or        more of the following rules/criteria:        -   Cases from patients who want to discuss results with the            expert.        -   Cases from patients who do not require a discussion.        -   Whether or not the case is from a referring doctor and/or a            type of doctor. For example, an expert may indicate “willing            to consult on elbow MRI scans with orthopedic surgeons but            not with family practice doctor.”        -   Legal cases where the submitter is an attorney working for a            defendant.        -   Legal cases where the submitter is an attorney working for a            plaintiff.        -   Cases submitted by insurance companies.    -   A blacklist of particular submitters. For example, an expert may        blacklist a submitter (or group of submitters) with whom the        expert has had an unsatisfactory interaction in the past. In an        embodiment, submitters that are blacklisted by an expert may not        see the expert on a list of available experts, and/or the system        may not select the expert for review of cases from submitters        that are blacklisted by the particular expert (even if, for        example, the expert matches other criteria established by the        submitter).    -   An indication of a schedule or availability. For example, an        expert may provide and manage a schedule of their availability        with the system such that they may not be matched to cases that        they may be unavailable to accomplish. In an embodiment, the        system may automatically determine an availability of an expert        based on past availability, past performance, a current case        load, and/or other characteristics of the expert.    -   An indication of an expected time to complete a report. For        example, an expert may provide, and/or the system may        automatically determine (based on, for example, past        performance, a current case load, and/or other characteristics        of the expert), an indication of an expected amount of time to        complete a report for a case. In an embodiment, multiple        expected times to complete various types of report may be        provided for a particular expert.

In an embodiment, and as described below in reference to FIG. 1B, thesystem may enable an expert to view particular cases (for example, caseswith which they are matched) such that the expert may evaluate the caseand decide whether or not to accept it.

At block 1215, a submitter may submit a case to the system. For example,the system may provide a user interface and/or computing device (asdescribed below in reference to FIGS. 2A-2B) through which the submittermay provide a case, including various files, images, information,characteristics, and/or the like. Examples of case submissions aredescribed below in reference to FIGS. 4-8. In an embodiment, block 1205is performed in conjunction with block 1215 such that criteria forselection of experts is associated with the current case beingsubmitted. Some submitters may have different criteria for each casesubmission and, thus, may provide those criteria along with the casesubmission.

At block 1220, the system may match a submitted case with particularexperts. For example, the system may automatically identify expertshaving characteristics appropriate to provide an opinion, report, orreview on the submitted case. In an embodiment, matching experts may bethose having all characteristics identified in the submitted case. Inanother embodiment, matching experts may be those having most, orparticular, characteristics identified in the submitted case. In anembodiment, in the event where no experts match the submitted case, thesubmitter may be provided with the option of altering thecharacteristics associated with the case so as to target, for example, agreater breadth of experts. In various embodiments, the system mayinclude rules and/or a rules engine that may perform matching of casesto experts.

At block 1225, the system may communicate the case to matching experts.In an embodiment, communication of the case may be performedautomatically once particular experts have been identified as havingcharacteristics matching the case. In various embodiments, a case may beprovided to one or many experts.

At block 1230, the experts that received the case may createreports/reviews in accordance with the case information and/orspecifications. In an example, the experts may create reports includingopinions of a medical image. As mentioned above, a report produced by anexpert may take any form (for example, written or verbal), and may beprovided via any medium.

At block 1235, the experts may provide their reports to the system. Anexpert may, in an embodiment, communicate their report to the systemthrough a user interface and/or computing device, as described below inreference to FIGS. 2A-2B.

At block 1240, the system may communicate the reports provided by theone or more experts to the submitter. In an embodiment, and as describedbelow in reference to FIGS. 2A-2B, the reports may be communicated tothe submitter through a user interface and/or computing device. In anembodiment, the submitter may evaluate various reports and select apreferred report. In another embodiment, the system may select aparticular report and provide that report to the submitter. For example,the system may evaluate the reports according to a set of rules and/orcriteria to determine a quality of the reports. Accordingly, the systemmay provide a report, or reports, to the submitter that meet a qualitythreshold. In an embodiment, the submitter may provide feedback to thesystem based on the quality of the reports. Such feedback may, forexample, be used by the system to score and/or rate the various expertsfrom whom reports were received.

FIG. 1B is a flow diagram illustrating another example method of anexpert opinion crowdsourcing system similar to the method of FIG. 1A,but including additional optional blocks, according to an embodiment ofthe present disclosure. As with FIG. 1A, the method of FIG. 1B may beperformed by the crowdsourcing server 100 (FIG. 2A) and/or othersuitable computing device. Depending on the implementation, the systemmay perform a method having more or fewer blocks than are shown, and/orthe blocks may occur in a different order and/or in parallel in order toaccomplish the methods and/or processes of the system.

At block 1205′ (where the prime indicator (′) in the reference numberindicates a block or a variation of a block (e.g., a combination ofmultiple blocks in a previous figure) having the same reference numberin a previous figure (e.g., FIG. 1A)), a submitter and one or moreexperts may provide and/or indicate various criteria and/orcharacteristics that may be used in matching a case of the submitter toone or more experts. The operation of this block is similar to theoperation of blocks 1205 and 1210 described above in reference to FIG.1A. Additionally, at block 1205′ the submitter may submit a case to thesystem. This operation is similar to the operation of block 1215described above in reference to FIG. 1A. Accordingly, the descriptionprovided above with reference to blocks 1205, 1210, and/or 1215 may beapplied to the present block 1205′.

In an embodiment, at optional block 1216, the submitter may provide apayment along with submission of the case. Alternatively, the submittermay provide a payment after receipt of a report (for example, afterblock 1240′ described below). The payment, or a portion of the payment,provided by the submitter may, as described below, be provided as acompensation to one or more experts who are matched and/or provide areport to the system and/or the submitter. In an embodiment, a portionof the payment provided by the submitter may be provided to the systemas a compensation for the use of the system.

At optional block 1217 the system may determine various characteristics,a complexity, and/or other information associated with the case. Thecharacteristics, complexity, and/or other information may be determinedfrom, for example, information provided by the submitter, metadataassociated with the case and/or one or more items of informationextracted from the case (for example, from headers, header files,metadata files or metafiles, notes or other textual content, imagerecognition, and/or the like). According to one embodiment, the systemmay determine the various characteristics/complexity automatically. Inan embodiment, the determined characteristics and/or complexity may beused by the system in addition to the characteristics/criteria providedby the submitter for selection of experts (as further described below).

In an embodiment, as described above, criteria for selection of expertsmay be associated with a current case being submitted. Some submittersmay have different criteria for each case submission and, thus, mayprovide those criteria along with the case submission. Additionally, inan embodiment blocks 1205′ and 1217 may be performed together such thatautomatically determined characteristics and submitter providedcharacteristics associated with the case may be provided along with thecase submission.

At block 1220′, the system may match a submitted case with particularexperts. For example, as described above, the system may automaticallyidentify experts having characteristics appropriate to provide anopinion or report on the submitted case. The operation of this block issimilar to the operation of block 1220 described above in reference toFIG. 1A. Accordingly, the description provided above may be applied tothe present block.

At optional block 1221, case information may be provided to matchingexperts. For example, various items of information and/orcharacteristics associated with the case may be provided to a matchingexpert such that the expert may determine whether the expert wants to,is able to, and/or is qualified to create a report and/or evaluate thecase. For example, an expert may want to not only see a list ofsubmitted cases (or exams), but the attributes of the case that wereextracted from metafiles, reports, and/or other records associated withthe case. In this embodiment, an expert may be enabled to decide if heor she wants to tackle the case compared to other listed cases.

At optional block 1222, match information may be provided to thesubmitter and/or the expert(s). For example, the system may provideinformation to one or more submitters and/or experts regarding acloseness (or strength) of a match (or a correlation) between a case anda particular expert(s). For example, the system may determine thatExpert A is a 70% match with a particular submitted case, while Expert Bis an 80% match with the same case. The closeness of a match (forexample, the percentages used in the previous example), may bedetermined based on, for example, a number of characteristics commonbetween the case and the expert. Match information may also be reportedin the form of a list of matching characteristics between the expert andthe case and/or qualifications of matched experts, among others. Inanother example, match information may include a list of matched expertsordered according to a particular ranking. Experts may be rankedaccording to a closeness of a match, an experience, a rating, and/orassociated qualification, just to name a few.

Such match information may be provided to the expert(s) such that theexpert(s) may make a determination regarding whether they desire and/orfeel qualified to take the case (for example, relative to more qualifiedexperts). Such match information may also be provided to the submittersuch that the submitter may make a determination regarding selection ofa particular expert based on the closeness of the match. In an example,when multiple experts are matched with a case (and/or accept a matchafter reviewing information associated with the case), the submitter maydesire to see how the experts rank in terms of the experts' attributesmatching with extracted characteristics from the case. For example, thesubmitter of a medical image for evaluation may want to see that thereare three general radiologists who seek to render an opinion, one juniorneuroradiologist, and one senior neuroradiologist with spectroscopyexpertise. The methods described above may similarly be applied in thematching of other types of experts including, for example, legal,ethical, and the like. As mentioned below, the system may determineprices for reports from particular experts based on matching informationand/or any other characteristics or criteria mentioned above. Such priceinformation may also be useable by the submitter to select a particularexpert or particular experts.

At block 1225′, the system may communicate the case to one or moredetermined experts. In an embodiment, communication of the case may beperformed automatically once particular experts have been determinedbased on, for example, expert characteristics matching casecharacteristics and/or selections/determinations made by the submitterbased on provided case and/or match information, as described above. Invarious embodiments, a case may be provided to one or many experts.Aspects of the operation of this block are similar to the operation ofblock 1225 described above in reference to FIG. 1A. Accordingly, thedescription provided above may be applied to the present block.

At block 1230′, the experts that received the case may create reports inaccordance with the case information and/or specifications, and theexperts may communicate the reports to the system. The operation of thisblock is similar to the operation of blocks 1230 and 1235 describedabove in reference to FIG. 1A. Accordingly, the description providedabove may be applied to the present block.

At optional block 1236, the experts that have provided their report tothe system may receive payment or compensation, as mentioned above.Alternatively, the experts may receive payment after their report isselected and/or accepted by the submitter. In various embodiments, thesystem may automatically determine prices associated with expertreports. Prices may be determined based on, for example, a complexity ofa case, a degree of matching between an expert and a case, an expert'sexperience and qualifications, and/or an expert's matching rank amongother matching experts, among others. In other embodiments, thesubmitter may provide a compensation amount per report or for a group ofreports. For example, a submitter may offer $15 per report, or possibly$15 for each report (up to a maximum of 3) from a reviewer with aparticular qualification, and compensation of $30 (for only a singlereviewer) for a review with a different (e.g., more specialized)experience. In one embodiment, the experts may set a minimumcompensation that they will accept for review of a case (possibly havingdifferent minimums for different case types/characteristics), or may bidon review of cases such that a lowest bidding reviewer wins the right tobe compensated for review of a case.

At block 1240′, the system may communicate the reports provided by theone or more experts to the submitter. The operation of this block issimilar to the operation of block 1240 described above in reference toFIG. 1A. Accordingly, the description provided above may be applied tothe present block.

In various embodiments, the system may include various other aspectsand/or features including, for example:

-   -   The system may provide tools that facilitate reading of cases,        such as within a browser wherein reviewers access case        information. For example, the system may include specialized        functionality that allows, for example, radiologists to        efficiently view and interpret medical images, such as those        that exist within Picture Archive and Communication Systems        (PACS).    -   The system may interface with, or be integrated into, Personal        Health Record systems (PHR) such that patients may make requests        directly from within a PHR and/or reports from experts may be        communicated to a PHR.    -   The system may interface with, or be integrated into, an        Electronic Medical Record system (EMR), Personal Health Record        systems (PHR), or Picture Archive and Communication System        (PACS) to allow unidirectional or bidirectional communication of        cases and/or reports managed by the expert opinion crowdsourcing        system.    -   The expert opinion crowdsourcing system may be linked to other        systems that experts may use for interpreting cases (for        example, to facilitate in the interpretation of cases). For        example, the system may be interfaced with a PACS system so that        a radiologist, serving as an expert, may utilize the PACS system        to interpret a case such as a medical imaging exam.    -   The system may require that submitters include contact        information for a patient's physician in the event that the        expert finds an important, but previously undiagnosed, condition        (for example, a cerebral aneurysm).

As mentioned above, in various embodiments, the system may performprocesses of rating experts. Ratings may be provided by, for example,submitters, other experts, and/or rules/criteria of the system. In someembodiments, ratings may be used, as mentioned above, as criteria formatching submitters, experts, and cases. Ratings of experts may bedetermined in a number of ways. For example, expert ratings may be basedon training, input from submitters, input from other experts, testing(for example, using known test cases), similarities of the expert'sreports with reports of other experts, follow-up with patients,comparison of the expert's prior reports with clinical or pathologicalfollow-up, and/or the like.

In various embodiments, aggregate rating values may be visible tosubmitters to select from a list of available experts and/or aggregateratings may be used as criteria for automated selection of experts (forexample, a submitter may indicate that experts have minimum rating of 4for the submitter's case). In an embodiment, the system mayautomatically stop sending cases to experts with ratings that fall belowa particular level or threshold.

Example Implementation Systems and Devices

FIGS. 2A-2B are block diagrams illustrating example computing systemsand/or devices that may be included in the expert opinion crowdsourcingsystem, according to embodiments of the present disclosure. Referring toFIG. 2A, the block diagram shows that the system may include a CaseSubmitter Computing Device 110, a Crowdsourcing Server 100, and anExpert Reader Computing Device 120. Further, the system may optionallyinclude, in some embodiments, a Medical Information System 210 a, aMedical Information System 210 b, and/or a Picture Archive andCommunication System (PACS) 211 b. Each of the components of the systemmay be in communication with any other component via, for example, wiredand/or wireless data connections. For example, medical informationsystems 210 a and 210 b, and PACS 211 b, may be in communication withcase submitter computing device 110 and/or expert reader computingdevice 120 via communication links 212 a, 212 b, and/or 212 c.Similarly, case submitter computing device 110, crowdsourcing server100, and/or expert reader computing device 120 may be in communicationwith one another via, for example, communication links 150. In variousembodiments, the system may include more or fewer components than areshown in FIG. 2A. Such communication links (for example, links 212 a,212 b, 212 c, and 150) may include one or more wired and/or wirelesscommunication networks, such as local area networks, wide area networks,cellular networks, the Internet, and the like.

In the example of FIG. 2A, the case submitter computing device 110 mayinclude a case submitter software module 111, a processor 181, a randomaccess memory (RAM) and/or storage 182, input/output devices 183(including, for example, a display, keyboard, and/or mouse), and/or anoperating system 184. The case submitter computing device 110 may beused by a submitter to communicate a case and/or case information, forexample, a medical case including medical information and/or variousmedical data (for example, medical images, reports, records, and thelike), to the crowdsourcing server 100. In an embodiment, the casesubmitter software module 111 may, as described below, includecomputer-executable instructions, or other software logic, that may beexecuted by, for example, the processor 181 to cause the case submittercomputing device 110 to, for example: receive and/or determinecharacteristics associated with a submitter and/or case, providesubmitter and case information (including characteristics) to thecrowdsourcing server 100, receive report information from thecrowdsourcing server 100, provide a user interface through which asubmitter may provide case information, and/or the like. In variousembodiments, the case submitter software module 111 may includeinstructions, or other software logic, to implement any other aspect orfunctionality of the system, as described herein. Provided caseinformation may be transmitted, by the case submitter computing device110, to the crowdsourcing server 100.

In various embodiments, case submitter computing device 110 may or maynot communicate with other systems, such as medical information system210 a and/or a Personal Health Care Record (PHR) system, a PictureArchive and Communication System (PACS), and/or an Electronic MedicalRecord (EMR) system. In some embodiments the functionality required tosubmit cases to crowdsourcing server 100 may be integrated into othersystems, such as a PACS, PHR or EMR, for example by incorporating casesubmitter software module 111 into these other systems. For example, insome embodiments the functionality discussed with reference to the casesubmitter computing device 110, including the case submitter softwaremodule 111, may be included in another computing device, such as anonline PHR system that allows members to submit cases for review byselecting experts via the crowdsourcing server 100.

Crowdsourcing server 100 may include a crowdsourcing server softwaremodule 101, a processor 181 a (similar to the processor 181), a randomaccess memory (RAM) and/or storage 182 a (similar to the RAM/storage182), input/output devices 183 a (similar to input/output devices 183),an operating system 184 a (similar to operating system 184), and/orvarious databases including, for example, a submitter database 103, anexpert database 104, a medical exam database 105, and/or an insurancedatabase 106. Similar to the case submitter software module 111described above, in an embodiment, the crowdsourcing server softwaremodule 101 may, as described below, include computer-executableinstructions, or other software logic, that may be executed by, forexample, the processor 181 a to cause the crowdsourcing server 100 to,for example: receive case information from submitters, receive expertinformation from experts, match cases to experts, provide cases toexperts, receive reports from experts, provide reports to submitters,rate experts, and/or the like. In various embodiments, the crowdsourcingserver software module 101 may include instructions, or other softwarelogic, to implement any other aspect or functionality of the system, asdescribed herein. Crowdsourcing server 100 may communicate with casesubmitter computing device 110 and/or expert reader computing device 120using any one or combination of wired and/or wireless communicationtechniques, such as local area networks, wide area networks, cellularnetworks, the Internet, email, and the like.

In various embodiments, the crowdsourcing server 100 may include orcommunicate with one or more databases or data structures. For example,submitter database (DB) 103 may hold information related to submitters,and expert DB 104 may hold information related to experts. Medical examDB 105 may hold information related to cases submitted. In someembodiments, medical exam DB 105 may hold information related to reportsof experts related to the submitted cases. Insurance DB 106 may holdinformation related to characteristics of medical insurance policies.For example, insurance DB 106 may including information related towhether or not an expert may charge a submitter for rendering an opinionon a submitted case. In other embodiments, insurance DB 106 may includeinformation on specific medical insurance covering patients associatedwith submitted cases. In other embodiments, insurance DB 106 may includeinsurance policies for which experts are contracted.

In various embodiments the various types of information described abovemay reside in databases other than the ones described. Additionally, thedatabases illustrated with reference to crowdsourcing server 100 maycomprise any other type of data structure for storing and/or organizingdata, including, but not limited to, relational databases (for example,Oracle database, mySQL database, and the like), spreadsheets, XML files,and text files, among others. The various terms “database,” “datastore,” and “data source” may be used interchangeably in the presentdisclosure. Further, in various embodiments the databases illustratedwith reference to crowdsourcing server 100 may be remotely located suchthat, for example, the crowdsourcing server 100 may accesses such datastructures via one or more networks.

Expert reader computing device 120 may include an expert software module121, a processor 181 b (similar to the processor 181), a random accessmemory (RAM) and/or storage 182 b (similar to the RAM/storage 182),input/output devices 183 b (similar to input/output devices 183), and/oran operating system 184 b (similar to operating system 184). The expertreader computing device 120 may be used by an expert, in variousembodiments described herein, to receive case information and/or providereports, among other things. For example the expert reader computingdevice 120 may communicate with crowdsourcing server 100 to provide theexpert access to cases. In some embodiments it may be used by the expertto view a case and/or create a report of his opinion. Similar to thecase submitter software module 111 described above, in an embodiment,the expert software module 121 may, as described below, includecomputer-executable instructions, or other software logic, that may beexecuted by, for example, the processor 181 b to cause the expert readercomputing device 120 to, for example: receive and/or determinecharacteristics associated with an expert, provide expert information(including expert characteristics) to the crowdsourcing server 100,receive case information from the crowdsourcing server 100, provide auser interface through which an expert may view case information,provide case information to an expert or other computing device, receivefrom and/or produce reports for experts, provide reports to thecrowdsourcing server 100, and/or the like. In various embodiments, theexpert software module 121 may include instructions, or other softwarelogic, to implement any other aspect or functionality of the system, asdescribed herein.

In various embodiments, the expert reader computing device 120 maycommunicate with other systems, for example medical information system210 b and/or a PHR system, a PACS (such as PACS 211 b), and/or an EHRsystem. For example, a case communicated to expert reading computingdevice 120 may be communicated to PACS 211 b so that a radiologistexpert may efficiently interpret a case and render an opinion in theform of a report. In some embodiments the functionality of expert readercomputing device 120 may be integrated into other systems or components,such as a PACS, PHR, and/or EHR. For example, in some embodiments theexpert software module 121 may be incorporated into one or more of theseother components.

Referring to FIG. 2B, the block diagram illustrates examples ofcomponents that may be present in a Medical Information System, such aseither of the optional medical information systems 210 a or 210 b ofFIG. 2A. In various embodiments, a medical information system mayinclude one or more of the components illustrated, or other systemsrelated to the management of medical information. Various devices andsubsystems illustrated in FIG. 2B may be connected to a network orvarious devices of the system (for example, via network 150 and/orcommunication links 212 a, 212 b, and/or 212 c) and may be incommunication with one or more of the components illustrated in FIG. 2A(for example, case submitter computing device 110, crowdsourcing server100, and/or expert reader computing device 120).

The medical information system 210 of FIG. 2B may include an MRI scanner220, a CT scanner 222, an Ultrasound scanner 224, a PACS database 230, aPACS image server 232, a PACS workstation 234, a radiology informationsystem 240, an electronic medical record system 250, a clinical labinformation system 260, a pathology information system 270, a personalhealth record 280, and/or a CAD system 290, among other components. TheMRI scanner 220 (among the other types of scanners), which may be usedto acquire MRI images from patients, may share the acquired images withother devices on the network 150. The network 150 may also be incommunication with one or more CT scanners 222 and/or ultrasoundscanners 224. The CT scanners 222 and Ultrasound scanners 224 may alsobe used to acquire images and, like the MRI scanner 220, may storeacquired images and/or share acquired images with other devices via thenetwork 150. Any other scanner or device capable of inputting orgenerating information that may be presented to a user (such as asubmitter or expert) as images, graphics, text, and/or sound may beincluded in the medical information system 210. Examples of other typesof devices may include angiography, nuclear medicine, radiography,endoscopy, pathology, dermatology, and/or the like.

Also connected to the network 150 may be the Picture Archiving andCommunications System (PACS) Database 230, PACS Image Server 232, andPACS workstation 234. PACS systems may be used for storage, retrieval,distribution, and presentation of images (such as those created and/orgenerated by the MRI scanner 220, CT Scanner 222, and/or UltrasoundScanner 224). Medical images may be stored in an independent format, anopen source format, and/or some other proprietary format. For example,images may be stored in the PACS system in a Digital Imaging andCommunications in Medicine (DICOM) format. The stored images may betransmitted digitally via the PACS system, which may reduce or eliminatethe need for manually creating, filing, and/or transporting film andfilm jackets.

The network 150 may also be connected to the radiology informationsystem (RIS) 240. The radiology information system 240 may be acomputerized data storage system that may be used by radiologydepartments to store, manipulate, and/or distribute patient radiologicalinformation.

Also attached to the network 150 may be the electronic medical record(EMR) system 250. The EMR system 250 may be configured to store and makeaccessible to a plurality of medical practitioners computerized medicalrecords. Also attached to the network 150 may be the clinical laboratoryinformation system 260. Clinical laboratory information system 260 maybe a software system which stores information created or generated byclinical laboratories. Also attached to the network 150 may be thedigital pathology system 270, which may be used to digitally manage andstore information related to medical pathology.

As shown in the embodiment of FIG. 2B, the personal health record (PHR)system 280 may also be coupled to the network. The PHR system 280 may beconfigurable by a particular patient in order to manage health recordsand data associated with the patient (and/or the patient's family orothers in the care of the patient). Also attached to the network 150 maybe the computer aided diagnosis system (CAD) 290 used to analyze imagesusing one or more computer aided techniques.

Other systems, devices, and/or components may also be in communicationvia the network 150. Such other systems, devices, and/or components mayinclude, for example, a 3D Processing System used to performcomputations on imaging information to create new views of theinformation (for example, 3D volumetric display, MultiplanarReconstruction (MPR), and Maximum Intensity Projection reconstruction(MIP)).

In various embodiments, other computing devices that store, provide,acquire, and/or otherwise manipulate medical data may also be coupled tothe network 150 and may be in communication with one or more of thedevices illustrated in the figures.

Example Communications among Submitters and Experts

FIG. 3 is a block diagram illustrating various examples of submittersand expert computing devices communicating with crowdsourcing server100, according to an embodiment of the present disclosure. In certainfigures herein, blocks may be labeled with an indicator of an individualor person that may control a computing device, such as a submitter or anexpert. Each such block may also include a computing system or device,such as one of the computing systems or devices illustrated withreference to FIG. 2A (for example, case submitter computing device 110and/or expert reader computing device 120). Similarly, certain figuresmay be labeled with indicators of computing systems or devices, ratherthan individuals or persons that operate the computing systems.Reference herein to an individual (such as a submitter or expert) or acomputing system or device (such as a case submitter computing device orexpert reader computing device) may refer to either the individual (forexample, a submitter or expert) and/or the computing system utilized bythe individual (for example, the computing device used by the submitteror the computing system used by the expert).

As shown in the example of FIG. 3, multiple submitter computing devices310 (including 310 a, 310 b, 310 c, 310 d, 310 e, and 310 f) may be incommunication with the crowdsourcing server 100 via the network 150. Asshown, the submitters may comprise various individuals that may submitmedical cases to the crowdsourcing server 100 for a variety of purposesand desired feedback options. For example, submitter 310 f may be apatient that may be submitting his/her own medical images in order toget an opinion or reading (or second, third, or fourth, among others) ofa radiology exam. In an embodiment, the patient may be submitting anexam that has already been read, for example, for another opinion. Inanother embodiment, the patient may be submitting an exam that has notbeen previously read to obtain one or more readings of their medicalimaging exam.

Submitter 310 e may comprise, for example, a third-party, such as aninsurance company, law firm, or the like. The third-party may request,for example, one or more expert reports on a medical case of a client(or adverse party) in order to prove or disprove an insurance claim orlegal case. Submitter 310 a may comprise, for example, a referringdoctor who may be requesting expert reports regarding a patient's casefor various purposes. For example, the referring doctor may berequesting reports to provide a further comfort level and/or guidance ina determined treatment course, and/or at the request of a patient.Submitter 310 b may comprise, for example, a radiologist that may beunsure of a particular diagnosis and who may desire other opinions inorder to increase the likelihood that the radiologist's final report isaccurate. These are just example motivations and purposes for providingmedical data to the crowdsourcing server 100; any other entity may be asubmitter and may submit medical data (or other types of data) for anyother purpose (although, as discussed herein, rules and/orcharacteristics established by experts and/or the crowdsourcing servermay limit which medical data is actually reviewed by particularexperts).

As shown in FIG. 3, and according to an embodiment, the system mayinclude expert computing devices 320 a and 320 b. The expert computingdevices 320 a and 320 b may be operated by experts, as determined by thecrowdsourcing server 100 and/or other entity. For example, the expertsthat control computing devices 320 a and 320 b may be radiologists, orpeople in training, that may desire further skill tuning that may beachieved by reviewing more difficult cases that may be available throughthe crowdsourcing server 100.

Additional Example Methods

FIGS. 4-8 are flow diagrams illustrating example methods or processes ofthe expert opinion crowdsourcing system in which cases from submittersare provided to experts, according to embodiments of the presentdisclosure. Turning to FIG. 4, the flow diagram illustrates an exampleflow of a medical case from a submitter 410 a, to a crowdsourcing server418, and then on to some of multiple experts 415 a-415 f. The example ofFIG. 4 illustrates several aspects of the system including, for example:

-   -   Cases may be associated with characteristics. In the example        illustrated, cases may be associated with a region or area of a        body imaged (in this example the spine) and the medical imaging        modality utilized (in this example, MRI). As described above,        characteristics, a complexity, and/or other information related        to a case may be provided by a submitter and/or determined from        case data automatically (for example, from metadata associated        with the case).    -   Experts may be associated with areas of expertise that correlate        with exam characteristics. In various embodiments, areas of        expertise of experts may be determined in a variety of ways        including, for example: by the experts themselves (for example,        the expert may provide information regarding their expertise),        by ratings of experts by others (for example, a credentialing        panel, other experts, feedback from submitters, and/or the        like), by testing the expert, by the expert's training, by a        specialty board associated with the expert, and/or by a license        associated with the expert, among others.    -   As mentioned above, various characteristics, or expertise, may        be associated with experts including, for example,        characteristics of cases the experts are willing to accept,        areas and/or modalities they are willing to accept, and the        like. In various embodiments, these characteristics and others        may be stored in one or more databases of the system.    -   The crowdsourcing server 418 may be configured to automatically        communicate cases where the characteristics of a case match the        characteristics, such as expertise, of the expert.

In the example of FIG. 4, a Spine MRI case is submitted by a submitter410 a, the case is determined to be compatible with particular expertsbased on the experts' characteristics, and the case is communicated to(or made available to) Experts A, C, and D as their areas of expertisematch the characteristics of the submitted exam.

As described above, characteristics, a complexity, and/or otherinformation associated with a case may be determined automatically bythe system. In an embodiment, the system may automatically determinecharacteristics associated with a medical exam from a DICOM header file,DICOM metafile, and/or other metadata or data included in the medicalexam and/or electronic medical record. For example, a DICOM modality ofa medical image or an image series may be automatically detected and/ordetermined from a DICOM metafile associated with the medical image orimage series. In another example, for a medial image, an anatomical areaof interest may be determined from a DICOM header and/or an ExamDescription (or other item of information) associated with the medicalimage. An Exam Description (or other item of information) may be a codedvalue (for example, a CPT-code (Current Procedural Terminology code) orSNOMED CT code) or non-coded value. In yet another example, a patientmay submit a report to the system (such as an imaging examinationclinical report, surgical report, or other expert report) for acrowd-sourced review, and the system may assess various codes associatedwith the report and/or text within or associated with the report todetermine characteristics of the report. In an embodiment, the systemmay use natural language processing to extract case characteristics froma report.

Automatically extracted and/or determined characteristics and/orcomplexity of a submitted exam may be used by the system in determiningparticular matching experts (as described above and below). Automaticdetermination of case characteristics may advantageously enable anunsophisticated patient user (or other user) to submit a case to thesystem and find a matching expert without manually characterizing thecase. For example, a patient submitter may not know that a Brain MRIideally should require a neuroradiologist or neurosurgeon expert, orthat a Sinus MRI may ideally require a Head and Neck radiologist or ENTsurgeon expert. However, by automatically extracting casecharacteristics from a submitted case, in these examples the system maynevertheless match the exam with appropriate specialist attributes ofthe expert reviewer. In an embodiment, automatically extracted and/ordetermined characteristics may be provided to either or both ofsubmitter(s) or expert(s) to review and/or approve (as described abovein reference to FIGS. 1A and 1B). Further, a confidence regarding acorrectness of automatically determined information may be determined bythe system and may be reported to submitter(s) and/or expert(s).

Regarding a case complexity determination mentioned above, in anembodiment the system may automatically categorize the case or reportbased on level of complexity of the case or report. Determining a levelof complexity may enable the system to match the case with particularexperts based on subspecialty qualifications, seniority, and/or otherrating systems. For example, a brain MRI may be directed to a general orneuroradiologist, while a Brain MRI with perfusion imaging andspectroscopy might be directed to a senior (or more qualified orexperienced) neuroradiologist. Alternatively the system may report toboth or either of the submitter or the matched expert(s) a list ofcandidates and the best matches based on a complexity and/or a ranking(as described above in reference to FIGS. 1A and 1B).

FIG. 5 is another flow diagram illustrating an example flow of adifferent case type: a PET of a brain. In this example, the case may betransmitted from the submitter 410 b to the crowdsourcing server 418,and then automatically, selectively may be made available to certainexperts. As shown in the illustration, the case may be communicated to,or made available to, Experts A, C, and F as their areas of expertisematch the characteristics of the submitted case.

FIG. 6 is another flow diagram illustrating an example transmission of acase to a crowdsourcing server 428, and also illustrating multiplesubmitters 420 a-420 e that may each be associated with variouscharacteristics. In the example of FIG. 6, each of the submitters isassociated with a “role”. For example, various roles of submitters mayinclude lawyer, doctor, patient, insurance company, and/or neurosurgeon,among others. In other embodiments, roles may be further subdivided, forexamples doctors may be further characterized by specialty, or lawyersmay be further subdivided by the role the lawyer is playing with regardto a case (for example, defendant vs. plaintiff).

In the embodiment of FIG. 6, expert characteristics may include types ofsubmitters from whom the expert is willing to accept cases. For example,the expert may specify particular submitter roles from which they acceptcases. In the example illustrated, such characteristics are listed asthe “Accepts” characteristic for each expert. For example, Expert A mayaccept cases from all types of submitters, Expert C may accept casesfrom patients and doctors, and Expert F may only accept cases submittedby doctors.

In the example illustrated, a patient 420 c may submit a Brain PET tocrowdsourcing server 428. The crowdsourcing server 428 may communicatethe case to experts that match, both in terms of the characteristics ofthe case and characteristics of the submitter. For example, because thesubmitter 420 c in this example is a patient that is submitting a BrainPET medical imaging exam, the exam is automatically matched to and madeavailable to Experts A and C.

FIG. 7 is another flow diagram illustrating an example transmission of acase to the crowdsourcing server 428. In the example, the submitter 420b is a doctor who is submitting a case including a Brain PET.Crowdsourcing server 428 automatically communicates the case toparticular experts that match, in this example Experts A, C, and F.

In other embodiments, other criteria may be used in the process ofmatching experts, submitters, and cases. For example, a location (suchas a particular state in the USA) associated with the case may be amongthe criteria used to choose experts. For example, in an embodiment onlyexperts who have a medical license in a state associated with a case maybe automatically chosen. In another example, insurance may be one of thecriteria used for matching submitters and experts. For example, thesystem may match cases to experts who are contracted with a patient'sinsurance company.

In another embodiment, the opposite may occur. For example, when doctorsare contracted with an insurance company, the contract may prohibit themfrom charging the patient for a second opinion. In a case where asubmitter desires to pay a doctor for a second opinion and the doctoragrees, the system may automatically match the case with experts who areNOT contracted with the patient's insurance company.

FIG. 8 is another flow diagram illustrating an example transmission of acase to a crowdsourcing servicer 438, and also illustrating associationof ratings with experts (as shown in 435 a-435 f), and minimum ratingrequirements provided by submitters (as shown in 430 a-430 e). Forexample, as mentioned above, submitters may provide criteria including aparticular expert rating, or minimum expert rating, when submitting acase. Expert ratings may be generated and/or updated by the system basedon a variety of factors including, for example:

-   -   Ratings provided by submitters (for example, after having        received a report from an expert).    -   Ratings provided by specific types of submitters (for example,        patients, other experts, or the like).    -   Ratings provided by a credentialing panel.    -   Ratings provided by other experts.    -   An expert's performance on a test or multiple tests.    -   An expert's level of training and/or experience.    -   A certification by, for example, a licensing board, a specialty        board, and/or the like.

In various embodiments, when an expert and/or expert report is ratedand/or feedback is provided by others, the system may weigh the ratingbased on attributes of the person providing the rating. For example, thesystem may determine an overall rating that takes into account theattributes of particular rating providers by, for example, placing agreater weight on a rating provided by a more qualified person orexpert. In an embodiment, the system may disclose the characteristics orattributes of specific persons providing ratings to, for example,submitters of a case. For example, a submitter may provide a case thatmay be matched with an expert. In reviewing the expert information priorto making a determination to send the case to the expert, the submittermay view not just a rating of the expert, but a breakdown of ratings ofthe expert, for example, “This expert is rated an 8/10 by generalradiologists, and a 6/10 by neuroradiologists.” Similarly, ratings forspecific report may be provided by the system.

In the embodiment of FIG. 8, each of the experts 435 a-435 f isassociated with a rating. In one embodiment, submitters may indicate aminimum rating of experts who may receive the case being submitted.Further, in an embodiment a submitter may indicate a particular numberof experts to review the case. For example, a submitter may indicate aminimum rating of 4, and 1 expert report, for a particular case. Thesystem (for example, the crowdsourcing server 438) may determine thattwo experts match the characteristics provided with the case. In thisexample, the system may automatically provide the case to the experthaving the highest rating.

In the example illustrated in FIG. 8, a submitter that is a doctor maysubmit a case that is a Brain PET, and may indicate that any expertsthat receive the case must have a minimum rating of 4. As shown in theexample, the case may be communicated to Experts A and F, as thoseexperts match in terms of the types of submitter from whom the expertwill accept cases, the characteristics of the exam, and the minimumrating of the expert that is acceptable to the submitter.

In various embodiments, any type of rating scale may be used by thesystem. For example, a rating scale may range from 1-5 (with either 1 or5 being the best), or 1-10 (with either 1 or 10 being the best), just toname two examples. Further, multiple ratings may be associated with eachexpert. For example, an expert may be associated with one rating thatmay be relevant to a particular type of submitter (for example, patientsubmitters) and another rating that may be relevant to a differentparticular type of submitter (for example, doctor submitters). Inanother example, an expert may be associated with one rating relevant toone area of expertise, such as imaging of the brain, and another ratingrelevant to another area of expertise, such as imaging of the chest.

FIGS. 9-10 and 11A-11B are flow diagrams illustrating example methods orprocesses of the expert opinion crowdsourcing system in which cases fromsubmitters are provided to experts, and reports from experts areprovided to submitters, according to embodiments of the presentdisclosure.

Turning to FIG. 9, a flow diagram is shown illustrating an exampleoverview of communications starting from transmission of a case by asubmitter to a crowdsourcing server, and finishing with the submitterreceiving reports from multiple experts. In the embodiment of FIG. 9, asubmitter 510 a may submit a case 530 to the crowdsourcing server 518and the case may be communicated to a number of experts 515 a, 515 b,and 515 c. The determination to transmit the case to Experts A, B, and Cmay be based on the functionality described above. In addition to thefunctionality described above, other criteria may be used toautomatically determine how cases are communicated to experts. Forexample, a submitter may indicate a maximum number of experts to which acase is to be communicated, and/or may indicate a desire to have thecase evaluated by multiple experts.

As shown in the embodiment of FIG. 9, experts 515 a, 515 b, and 515 c,each provide reports back to the crowdsourcing server 518, which maythen automatically communicate the reports to the submitter 510 a ormake the reports available to the submitter 510 a. In some embodiments,the experts 515 a-515 c may communicate the reports directly to thesubmitter 510 a.

FIG. 10 illustrates an example including components similar to those ofFIG. 9. In the example of FIG. 10, however, the case 530 submitted bythe submitter 520 a, as well as the reports provided by experts 525a-525 c, are anonymized by the crowdsourcing server 518. In someembodiments, cases may be anonymized, for example such that experts maybe unaware of an identity of a submitter and/or a patient associatedwith a case. Similarly, in some embodiments, reports may by anonymizedsuch that, for example, a submitter and/or patient associated with acase may be unaware of an identity of an expert.

In some embodiments, the crowdsourcing server may track information thatmay allow the identity of the case, expert, and/or submitter to bedetermined. For example, a patient submitter may submit a case andindicate that it is to be anonymized. In addition, an expert may requirethat a report be anonymized so that the patient and/or submitterassociated with the report are not identified. In another example, theidentity of the expert creating the report may be hidden.

After the submitter receives a report, the submitter may request theidentity of an expert so that the submitter may communicate further. Ifthe expert agrees to be identified to the patient, and the patientagrees to be identified to the expert, then the crowdsourcing server 518may provide the identifying information to each party and/or providefunctionality that may allow the two parties to communicate (forexample, to create a doctor-patient relationship).

In an embodiment, an expert may provide criteria that may require thatcases that they accept include contact information for a physiciancaring for a patient associated with a submitted case so that the expertmay contact the patient's physician if the expert finds a significantabnormality.

As mentioned above, FIGS. 11A and 11B are sequential flow diagramsshowing an example method of the expert opinion crowdsourcing system,according to an embodiment. FIGS. 11A and 11B show an example methodthat may allow for rating of reports of experts. In an embodiment,rating of expert report may, for example, put various experts in a sortof competition to provide the best and/or most accurate reports. FIG.11A demonstrates initial steps in the embodiment, and FIG. 11Bdemonstrates additional steps. As in some other embodiments, submittersand experts may indicate criteria that may be used to determine how acrowdsourcing server 600 automatically matches experts, submitters, andcases.

In the example of FIG. 11A, a submitter 620 a may communicates a case630 to the crowdsourcing server 600. As described in other embodiments,the crowdsourcing server 600 may automatically choose one or moreexperts to which the case is to be communicated.

In this embodiment, the case is then anonymized by the crowdsourcingserver 600, and then the anonymized case 631 is communicated to matchingexperts 625 a, 625 b, and 625 c. Depending on the embodiment,anonymization may take different forms. For example, in one embodiment,anonymization may comprise removing any personally identifiableinformation from the case 630, such as a patient's name, contactinformation, and/or the like. In some embodiments, anonymization mayinclude removing such data from actual medical images, for example,medical images that have personally identifiable information included inthe images. In some embodiments, the crowdsourcing server 600 mayinclude image analysis capabilities that allow automatic identificationand obfuscating (or removal) of personal information of a patient. Insome embodiments, anonymization may further comprise removal ofinformation regarding a source of the images (for example, an imagingcenter), information regarding a referring doctor that originallyrequested the medical images, information regarding a location of apatient, doctor, and/or imaging center, and/or any other information. Inanother embodiment, the case may not be anonymized.

In an embodiment, the each of the experts 625 a-625 c may be notified ofthe experts 625 a-625 c that have been matched to the particular case630. In other embodiments, each of the experts may have no knowledge ofother experts that have been matched to a particular case.

In the example of FIG. 11A, experts 625 a, 625 b and 625 c, createreports 640 a, 640 b, and 640 c, which may be communicated to thecrowdsourcing server 600. In some embodiments, the crowdsourcing server600 may provide a graphic user interface that experts may use to createreports directly on the crowdsourcing server 600.

Continuing on to FIG. 11B, the reports from various experts may beanonymized and communicated to one or more “feedback entities,” such asexperts (e.g., experts that provided reports and/or other experts), thesubmitter, and/or any other entity from which feedback on reports may bedesired (e.g., individuals that are neither the submitter nor an expertthat provides a report), for evaluation and/or feedback. For example, acompilation of reports received from multiple experts may be provided toone or more experts, wherein a source of individual reports may not beidentifiable. In the example of FIG. 11B, a compiled report 650, whichmay include reports from each of the experts 625 a, 625 b, and 625 c,may be communicated from the crowdsourcing server 600 to each of theexperts. The compiled report 650 may simply include each of the expertreports in their entireties, or may include portions of the reports,such as in a table or chart format configured for easier review of themultiple reports. In other embodiments, the reports may not beanonymized before being communicated to the experts.

In another embodiment, the group of experts receiving the reports may bedifferent than the experts that created the reports. For example, in oneembodiment radiologists without specialized training in neuroradiologymay view neuroradiology cases and provide reports, while a group ofexperts evaluating (and/or providing feedback on) the reports may berestricted to neuroradiologists (for example, radiologists withsubspecialty training in neuroradiology).

In another embodiment, the group of experts evaluating the reports mayinclude one or more of the group of experts that created the report aswell as one or more other experts.

In yet another embodiment, the people rating and/or providing feedbackon the reports may not be experts. For example, in one embodiment peoplewho are not physicians may provide feedback concerning and/or ratemedical reports based on how clearly each expert's reports communicatedtheir opinions to people without medical training. In anotherembodiment, submitters may provide feedback concerning and/or rate theexperts' reports.

The group evaluating the reports may then vote, applying a rating toeach of the reports. For example, in one embodiment each of the votingexperts (the experts that have access to reports provided by otherexperts for the purpose of providing feedback on the reports) votes forthe report or diagnosis that they think provides the most accurateevaluation of the case. In other embodiments, other feedback and/orvoting processes may be performed by the voting experts. The votes (660a, 660 b, and 660 c) from the experts may be communicated to thecrowdsourcing server 600. In an embodiment, the crowdsourcing server 600may provide a user interface that may allow the experts evaluating thereports to view the case and vote on the reports, for example via a webbrowser interface.

Based on the votes 660 a-660 c received by the crowdsourcing server 600,experts may receive points based on the ratings of their report. Thesepoints may be used to determine a “best” (or “winner”) report ordiagnosis which may be made available to the submitter. Such points maybe used to determine and/or update an expert's rating and/or raking,such as with reference to a particular specialty associated with thecase and/or an overall rating for the expert. In an embodiment, thisrating and/or ranking may be made available to others, such assubmitters, who may use it as a criterion for selection of experts, forexample for additional case submissions.

In an embodiment, the crowdsourcing server 600 may generate a compositereport 670 with may include results of the voting and/or the variousreports generated by the experts 625 a-625 c. For example, the compositereport 670 may include each of the individual reports listed in an orderbased on the voting. In another example, the composite report 670 mayinclude portions of each of the reports based on the voting.

Additional Embodiments

In an embodiment, the crowdsourcing server may post (for example, on awebsite) a request and/or challenge for medical data that may be usefulto others. For example, the expert opinion crowdsourcing system maycollect pathologically proven cases that have been reviewed by one ormore experts. In another example, the crowdsourcing server may post achallenge, such as “In the next 60 days, we want to build a file of themost common imaging findings a first year resident should know how torecognize before they start taking call,” in order to obtain data from avast audience of experts that may be useful for a particular purpose(for example, educating medical trainees in this example).

In another embodiment, the expert opinion crowdsourcing system includesa publicly accessible user interface (generated by, for example, asoftware module of the crowdsourcing server) that may include, forexample, ratings associated with experts.

In an embodiment, experts receiving positive feedback or ratings from,for example, submitters, other experts, and/or other feedback entities,may receive rewards. For example, an expert receiving a highest ratingfrom among a group of experts reviewing/providing a report on a case mayreceive an award. Examples of reward may include monetary rewards,notations, badges, discounts, notoriety, and/or the like. Such rewardsmay be included on (or in), for example, a user interface of the system.

Example Computing System Components and Operation

As described above, FIG. 2A illustrates various components of acrowdsourcing server 100, a case submitter computing device 110, and anexpert reader computing device 120. Each of these computing systems ordevice may, in various embodiments, take various forms. For example,each of the computing systems or devices may include any combination ofthe components and/or functionality described below, as well as othercomputer hardware and/or software. The hardware will be discussed withreference to a “computing system,” which could apply to any of thecrowdsourcing server 100, the case submitter computing device 110,and/or the expert reader computing device 120.

In one embodiment, the computing system may be a computer workstationhaving one or more software modules (for example, modules 101, 111,121). In other embodiments, software modules may reside on othercomputing devices, such as a web server or other server, and the userdirectly interacts with a second computing device that is connected tothe web server via a computer network.

In one embodiment, the computing system comprises a server, a desktopcomputer, a workstation, a laptop computer, a mobile computer, asmartphone, a tablet computer, a cell phone, a personal digitalassistant, a gaming system, a kiosk, an audio player, any other devicethat utilizes a graphical user interface, including office equipment,automobiles, airplane cockpits, household appliances, automated tellermachines, self-service checkouts at stores, information and otherkiosks, ticketing kiosks, vending machines, industrial equipment, and/ora television, for example.

The computing system may run an off-the-shelf operating system such asWindows, Linux, MacOS, Android, or iOS. The computing system may alsorun a more specialized operating system which may be designed for thespecific tasks performed by the computing system.

The computing system may include one or more computing processors (forexample, processors 181, 181 a, and 181 b). The computer processors mayinclude central processing units (CPUs), and may further includededicated processors such as graphics processor chips, or otherspecialized processors. The processors generally may be used to executecomputer instructions based on the software modules to cause thecomputing device to perform operations as specified by the modules. Themodules may include, by way of example, components, such as softwarecomponents, object-oriented software components, class components andtask components, processes, functions, attributes, procedures,subroutines, segments of program code, drivers, firmware, microcode,circuitry, data, databases, data structures, tables, arrays, andvariables. For example, modules may include software code written in aprogramming language, such as, for example, Java, JavaScript,ActionScript, Visual Basic, HTML, Lua, C, C++, or C#. While “modules”are generally discussed herein with reference to software, any modulesmay alternatively be represented in hardware or firmware. Generally, themodules described herein refer to logical modules that may be combinedwith other modules or divided into sub-modules despite their physicalorganization or storage.

The computing system may also include memory (for example, RAM/storage182, 182 a, and 182 b). The memory may include volatile data storagesuch as RAM or SDRAM. The memory may also include more permanent formsof storage such as a hard disk drive, a flash disk, flash memory, asolid state drive, or some other type of non-volatile storage.

The computing system may also include or be interfaced to one or moreperipheral devices or input/output devices (for example, input/outputdevices 183, 183 a, and 183 b) that provide and/or receive informationto/from the users. Peripheral devices may include one or more displaydevices that may include a video display, such as one or morehigh-resolution computer monitors, or a display device integrated intoor attached to a laptop computer, handheld computer, smartphone,computer tablet device, or medical scanner. In other embodiments, thedisplay device may include an LCD, OLED, or other thin screen displaysurface, a monitor, television, projector, a display integrated intowearable glasses, or any other device that visually depicts userinterfaces and data to viewers.

The peripheral devices may also include or be interfaced to one or moreinput devices which receive input from users, such as a keyboard,trackball, mouse, 3D mouse, drawing tablet, joystick, game controller,touch screen (e.g., capacitive or resistive touch screen), touchpad,accelerometer, video camera and/or microphone.

The computing system may also include one or more interfaces which allowinformation exchange between computing system and other computers andinput/output devices using systems such as Ethernet, Wi-Fi, Bluetooth,as well as other wired and wireless data communications techniques.

The modules of the computing system may be connected using a standardbased bus system. In different embodiments, the standard based bussystem could be Peripheral Component Interconnect (“PCI”), PCI Express,Accelerated Graphics Port (“AGP”), Micro channel, Small Computer SystemInterface (“SCSI”), Industrial Standard Architecture (“ISA”) andExtended ISA (“EISA”) architectures, for example. In addition, thefunctionality provided for in the components and modules of thecomputing system may be combined into fewer components and modules orfurther separated into additional components and modules.

The computing system may communicate and/or interface with other systemsand/or devices. In one or more embodiments, the computer system may beconnected to a computer network 150. The computer network 150 may takevarious forms. It may be a wired network or a wireless network, or itmay be some combination of both. The computer network 150 may be asingle computer network, or it may be a combination or collection ofdifferent networks and network protocols. For example, the computernetwork 150 may include one or more local area networks (LAN), wide areanetworks (WAN), personal area networks (PAN), cellular or data networks,and/or the Internet.

Depending on the embodiment, certain acts, events, or functions of anyof the processes or algorithms described herein may be performed in adifferent sequence, may be added, may be merged, and/or may be left outaltogether (for example, not all described operations or events arenecessary for the practice of the process or algorithm). Moreover, incertain embodiments, operations or events may be performed concurrently,for example, through multi-threaded processing, interrupt processing, ormultiple processors or processor cores or on other parallelarchitectures, rather than sequentially.

The various illustrative logical blocks, modules, routines, andalgorithm steps described in connection with the embodiments disclosedherein may be implemented as electronic hardware, computer software, orcombinations of both. To clearly illustrate this interchangeability ofhardware and software, various illustrative components, blocks, modules,and steps have been described above generally in terms of theirfunctionality. Whether such functionality is implemented as hardware orsoftware depends upon the particular application and design constraintsimposed on the overall system. The described functionality may beimplemented in varying ways for each particular application, but suchimplementation decisions should not be interpreted as causing adeparture from the scope of the disclosure.

The steps of a method, process, routine, or algorithm described inconnection with the embodiments disclosed herein may be embodieddirectly in hardware, in a software module executed by a processor, orin a combination of the two. A software module may reside in RAM memory,flash memory, ROM memory, EPROM memory, EEPROM memory, registers, harddisk, a removable disk, a CD-ROM, or any other form of a non-transitorycomputer-readable storage medium. An example storage medium may becoupled to the processor such that the processor may read informationfrom, and write information to, the storage medium. In the alternative,the storage medium may be integral to the processor. The processor andthe storage medium may reside in an ASIC. The ASIC may reside in a userterminal. In the alternative, the processor and the storage medium mayreside as discrete components in a user terminal.

Conditional language used herein, such as, among others, “can,” “could,”“might,” “may,” “for example,” and the like, unless specifically statedotherwise, or otherwise understood within the context as used, isgenerally intended to convey that certain embodiments include, whileother embodiments do not include, certain features, elements and/orsteps. Thus, such conditional language is not generally intended toimply that features, elements and/or steps are in any way required forone or more embodiments or that one or more embodiments necessarilyinclude logic for deciding, with or without author input or prompting,whether these features, elements and/or steps are included or are to beperformed in any particular embodiment. The terms “comprising,”“including,” “having,” and the like are synonymous and are usedinclusively, in an open-ended fashion, and do not exclude additionalelements, features, acts, operations, and so forth. Also, the term “or”is used in its inclusive sense (and not in its exclusive sense) so thatwhen used, for example, to connect a list of elements, the term “or”means one, some, or all of the elements in the list.

Conjunctive language such as the phrase “at least one of X, Y and Z,”unless specifically stated otherwise, is to be understood with thecontext as used in general to convey that an item, term, etc. may beeither X, Y, or Z, or a combination thereof. Thus, such conjunctivelanguage is not generally intended to imply that certain embodimentsrequire at least one of X, at least one of Y, and at least one of Z toeach be present.

While the above detailed description has shown, described, and pointedout novel features as applied to various embodiments, it may beunderstood that various omissions, substitutions, and changes in theform and details of the devices or processes illustrated may be madewithout departing from the spirit of the disclosure. As may berecognized, certain embodiments of the inventions described herein maybe embodied within a form that does not provide all of the features andbenefits set forth herein, as some features may be used or practicedseparately from others. The scope of certain inventions disclosed hereinis indicated by the appended claims rather than by the foregoingdescription. All changes which come within the meaning and range ofequivalency of the claims are to be embraced within their scope.

1-13. (canceled)
 14. A computing system comprising: a storage deviceconfigured to store electronic software instructions; and one or morecomputer processors configured to execute the stored softwareinstructions to cause the computing system to: receive a case includingassociated case characteristics, the case provided by a submitter, thecase characteristics including a submitter type; receive expertinformation associated with experts, the expert information includingexpert characteristics associated with each respective expert, theexpert characteristics including indications of accepted submittertypes; match the case to one or more experts based on the casecharacteristics and the expert characteristics, wherein the submittertype associated with the case matches the indications of acceptedsubmitter types associated with the one or more matched experts; andprovide the case to the matched experts.
 15. The computing system ofclaim 14, wherein the case includes one or more medical images, andwherein the experts include physicians having expertise in review ofrespective types of medical images.
 16. The computing system of claim14, wherein the one or more computer processors are further configuredto execute the stored software instructions to cause the computingsystem to: receive reports from the matched experts, the reportsincluding evaluations of the provided case; provide at least one of thereceived reports to the submitter; receive, from the submitter, a ratingassociated with the at least one provided report; associate the ratingwith an expert that provided the at least one provided report.
 17. Thecomputing system of claim 16, wherein the expert characteristics includeratings associated with respective experts, the case characteristicsinclude a minimum expert rating, and matching the case to the one ormore experts includes determining that a rating associated with aparticular matched expert satisfies the minimum expert rating.
 18. Thecomputing system of claim 16, wherein the submitter type includes atleast one of: a patient, a patient that wants to discuss results, apatient that does not require a discussion, a doctor, a type of doctor,a referring doctor, a lawyer, an attorney working for a defendant, anattorney working for a plaintiff, or an insurance company.
 19. Acomputer-readable, non-transitory storage medium storing computerexecutable instructions that, when executed by a computer system,configure the computer system to perform operations comprising:receiving a case including associated case characteristics, the caseprovided by a submitter, the case characteristics including anindication that the submitter desires to be able to contact an expert;receiving expert information associated with experts, the expertinformation including expert characteristics associated with eachrespective expert, the expert characteristics including indicationswhether respective experts are willing to be contacted by the submitter;matching the case to one or more experts based on the casecharacteristics and the expert characteristics, wherein casecharacteristics associated with the one or more matched expertsindicated that the respective experts are willing to be contacted by thesubmitter; and providing the case to the matched experts.
 20. Thenon-transitory computer-readable medium of claim 19, the operationsfurther comprising: receiving, from each of the matched experts, reportsbased on the provided case; generating a composite report based on thereceived reports; and providing the composite report to the submitter.21. The non-transitory computer-readable medium of claim 20, wherein thecomposite report is generated based on ratings associated with each ofthe reports.
 22. The non-transitory computer-readable medium of claim21, wherein the ratings associated with each of the reports are providedby the matched experts.